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New  Software  Framework  for  Solving  Hyperbolic 
Conservation  Laws 


®  This  presentation  discusses  the  construction  of  the  new  software 
framework  that  supports 

a  pluggable  multi-physics, 
a  hybrid  parallelism  for  HPC,  and 
a  productive  work  flows, 

to  deliver  analyzed  results  by  using  high-fidelity  solutions  of 
hyperbolic  conservation  laws. 

®  The  new  software  framework  is  called  SOLVer  CONstructor,  i.e., 
SOLVCON;  it  is  a  platform  to  construct  PDE-solving  codes. 
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SOLVCON  Framework 


Design  of  the  Software  Framework 


Python  Programming  Language 

®  Python  enables  high-level  constructs: 
a  Pluggable  multi-physics, 
a  Automatic  hybrid  parallelism, 
a  Parallel  I/O  and  in  situ  analysis/visualization. 

®  Python  is  a  dynamically-typed  programming  language. 

a  Support  multiple  programming  paradigms:  procedural  (like  Fortran  or 
C),  object-oriented  (like  C++  or  Java),  and  functional  (like  Lisp  or 
Scheme). 

a  Realize  high-level  construct:  type  registries,  plug-ins,  etc. 

®  Python  is  designed  to  glue  multiple  programming  languages  together. 

a  Use  CUDA,  C,  pthread,  and  MPI  simultaneously. 

®  Python  is  suitable  to  extend  SOLVCON's  functionalities: 

a  100+  packages  in  standard  library  and  13000+  3rd-party  packages, 
a  Wrappers  to  many  existing  toolkits:  VTK,  netCDF,  MPI,  etc. 
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Design  of  the  Software  Framework 

Python  and  C/C++ 


»  Python  and  C  -  Ctypes 

a  Compile  the  C  code  as  a  shared  object 

a  Python/ctypes  cannot  read  in  c  header  files  so  some  of  the  contents  of 
the  header  files  will  need  to  be  rewritten  in  python,  function  definitions 
do  not  need  to  be  rewritten. 

a  Load  the  library  from  python  using  ctypes  module  and  save  to  an  object 
a  This  object  contains  each  subroutine  from  the  shared  object  file. 

9  Python  and  C++  -  Boost 

a  This  is  accomplished  by  writing  wrappers  to  the  C++  classes 
a  Does  not  require  any  changes  to  the  C++  code 
a  I  have  not  used  this  before  as  python  handles  the  objects 
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SOLVCON  Framework 


Two-Loop  Structure 


»  All  time-accurate 
finite-volume  methods  contain 
two  loops. 

9  Temporal  loop  time-marches 
for  temporal  integration, 
a  Spatial  loops  iterate  over 
elements  to  calculate  flux. 

»  These  two  loops  form  the 
basic  structure  of  SOLVCON. 


Design  of  the  Software  Framework 


dT”  Begin/Initialization 

I 


Temporal  Loop  (Time-March): 
Current  Time  and  Step,  Time  Increment 
Execution  Controller 


Spatial  Loops  (Sub-Loops): 
Flux  Phase  1 
Flux  Phase  2 

Flux  Phase  N 
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SOLVCON  Framework 


Design  of  the  Software  Framework 

Five-Layer  Architecture 


a  Code  is  organized  by  using  Python  modules  (blue  solid  boxes), 
a  A  module  depends  only  on  other  modules  in  the  same  layer  or  in  the 
lower  layers. 

a  The  two-loop  structure  is  hosted  in  the  execution  layer. 
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SOLVCON  Framework 


Code  Development  for  SOLVCON 


®  The  driving  scripts  are  the  highest-level  construct  of  SOLVCON. 

a  A  driving  script  must  create  a  Case  object  and  call  its  (i)  init(),  (ii) 
runQ,  and  (iii)  cleanupQ  methods. 

a  The  Case  object  represents  the  overall  execution  flow  of  the 
simulation,  and  contains  the  temporal  loop. 

®  The  driving  scripts  can  specify  logic  to  the  simulations  in  addition  to 
parameters. 

a  Anything  higher  than  the  foundation  layer  (the  lowest  layer)  can  be 
replaced  by  code  written  in  driving  scripts, 
a  Including  but  not  limited  to  Case,  Solver,  BC  classes,  Hook  and 
Anchor  classes. 

®  SOLVCON  does  not  use  input  files,  but  uses  driving  scripts  instead 
(because  Python  code  needs  no  explicit  compilation). 

4  a  >  4  g 


Re  I  easef*6 1st  ri'j? u  t  ion  *l?n  I  frn  iteS 


osu 


•0  0,0 

8/19 


SOLVCON  Framework 


Code  Development  for  SOLVCON 


Calculation 

»  Solver 

a  This  is  a  generic  hyperbolic  non-linear  solver  that  has  been  optimized 
for  both  CPU  and  GPU  and  has  been  simplified  for  linear  equations, 
a  This  is  accomplished  through  the  use  of  pre- processors  that 
optimize/simplified  certain  portions  of  the  code 
a  Through  the  use  of  functional  pointers  these  routines  will  call  the 
required  Jacobian  subroutines, 
a  Constitutive  Equations 

a  These  subroutines  are  called  by  the  solver  schemes  and  calculate  the 
Jacobian  and  fluxes 

a  SOLVCON  has  built  in  support  for  Euler  and  linear  solvers, 
a  New  physics  can  be  added  by  creating  new  Jacobian  routines  and  by 
modifying  certain  parts  of  the  default  case  and  solver  codes, 
a  This  is  accomplished  through  inheritance 
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Code  Development  for  SOLVCON 


Employment 


»  batch:  Used  to  submit  parallel  jobs  on  a  cluster.  Built  in  support  for 
Torque 

»  io:  Reads  in  the  mesh  and  writes  the  VTK  output  files 

a  Input  capable  of  reading  in:  Gambit  Neutral  and  Genesis/Exodus, 
a  Output  capable  of  writing:  VTK  in  both  serial  and  parallel,  ASCII  and 


binary. 


®  visuaLvtk:  Provides  real  time  visualization  and  access  to  VTK 
through  the  VTK  python  wrappers. 

®  command:  Provides  the  infrastructure  for  command  line  arguments 
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Execution 


SOLVCON  Framework 


Code  Development  for  SOLVCON 


»  hook:  Allocates  locations  where  the  user  can  insert  code  into  the 
temporal  loop.  Code  inserted  here  can  run  in  serial  mode  only 

9  anchor:  Similar  to  the  hook  but  the  code  is  inserted  into  spatial  loop. 

Code  inserted  here  runs  on  each  compute  node. 

9  case:  Provides  the  basic  helper  subroutines  to  support  the  solver  such 
as: 

a  cfl  calculations 
a  Convergence  checks 

a  I/O  post  processing  support  such  as  track  results  along  a  line  or  at  a 
particular  point 

9  solver:  Defines  the  structure  of  the  main  program, 
a  Defines  the  main  data  structure 
a  Initializes/creates  the  data  in  the  structure 
a  Provides  the  routine  to  be  called  in  the  marching  routine 

<  □  ►  <9 
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Code  Development  for  SOLVCON 

Boundary-Condition  Treatments 


»  SOLVCON  uses  ghost  cells  to  treat  boundary  conditions  (BC). 

a  BC  treatments  depend  on  (i)  numerical  algorithms,  (i)  physical  models, 
and  (iii)  mesh  data  structures. 

®  SOLVCON  decouples  BC  treatments  from  numerical  algorithms, 
a  The  BC  class  hierarchy  is  used  to  hold  the  code. 

®  A  BC  treatment  is  a  spatial  sub-loop  that  iterates  over  only  boundary 
cells. 


ConcreteSolver (Solver) : 
def  bound_flux()  : 

for  be  in  self.bclist: 

be  ,bound_flux() - 

def  bound_gradient () : 
for  be  in  self.bclist: 
be  .  bound_gradient  ()  - 


ConcreteBC (BC) : 

def  bound_flux()  : 

-*  def  bound^gradient  ()  : 
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Code  Development  for  SOLVCON 


Boundary-Conditions 


9  Boundary  conditions  can  be  defined  by  either  the  solver  or  the  physics, 
a  Boundary  conditions  specified  by  the  solver  are  generic  and  are 
applicable  to  all  physics.  These  boundary  conditions  are: 
a  Non-reflecting 
a  Periodic 

9  Boundary  conditions  specified  by  the  physics  are  only  available  to  the 
physics  that  creates  them.  Some  examples  are: 
a  Non-Viscous  wall 
a  Pressure  inlet/outlet 
a  Non-Conducting 
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Code  Development  for  SOLVCON 


Foundation 

®  Mesh 

a  domain:  oversees  the  domain  decomposition  using  Metis  or  Scotch  and 
distributes  the  domains  over  the  network 
a  block:  Provides  the  data  structures  for  the  unstructured  mesh 
®  distributed  parallel 

a  rpc:  Remote  Procedure  call  and  inter-process  communication 
a  mpy:  Python  wrappers  to  MPI 

a  connection:  Remote  connections  and  communications  between  nodes 
®  Shared  parallel 

a  scuda:  A  wrapper  to  the  CUDA  shared  libraries  through  ctypes 
a  mthread:  Multi-threading  though  pthreads,  OpenMP,  MPI 

®  utility 

a  conf:  Info  about  the  configuration  of  SOLVCON 
a  gendata:  Generic  data  structure 
a  dependency:  Manages  the  external  shared  libraries 
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Code  Development  for  SOLVCON 

Currently  Supported  2/3D  Primitive  shapes 


a  Two-dimensional  elements: 


o 


triangle 

9  Three-dimensional  elements: 


quadrilateral 


pyramid 


Re  I  easef*6 1st  ri'j? u  t  !on  *l?n  I  frn  iteS 


osu 


>0  0,0 

15/19 


SOLVCON  Framework 


Code  Development  for  SOLVCON 

Data  Structures  of  Unstructured  Meshes 


a  Three  types  of  entities:  nodes,  faces,  and  cells, 
a  The  spatial  domain  of  interest  is  covered  by  non-overlapping  cells, 
a  Two  sets  of  arrays  define  the  meshes. 


Connectivity 

a  clnds:  nodes  in  each  cell, 
a  clfcs:  faces  in  each  cell, 
a  fends:  nodes  in  each  face, 
a  feels:  cells  related  by  each 
face. 


Geometry 

a  nderd:  coordinates  of  each 
node. 

a  feend:  center  of  each  face. 

a  fcnml:  unit  normal  vector  of 
each  face. 

a  fcara:  area  of  each  face, 
a  elend:  center  of  each  cell, 
a  clvol:  volume  of  each  cell. 
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Code  Development  for  SOLVCON 

On-the-Fly  Analysis 


»  Solution  processing  is  not  part  of  the  numerical  algorithms. 

»  SOLVCON  uses  the  callback  mechanism  to  separate  the  supportive 
functionalities  from  numerical  algorithms  and  physical  models, 
a  Hook:  The  outer  temporal  loop, 
a  Anchor:  The  inner  spatial  loops. 

»  Example  1:  Initial  condition. 

a  SOLVCON  calls  the  preloopO  method  before  entering  the  temporal 
loop. 

»  Example  2:  Calculate  physical  quantities. 

a  SOLVCON  calls  the  postmarchO  method  after  finishing  all  spatial 
loops  for  each  time  step. 

»  The  analysis  code  can  be  packaged  with  solver  kernels. 
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Code  Development  for  SOLVCON 


Coding  for  In  Situ  Visualization 


»  SOLVCON  directly  calls  external  visualizing  libraries  by  using  Python. 

a  Currently  support  VTK. 
a  VTK  interface  is  provided  in  SOLVCON. 

a  Provide  one-way  data  converter  from  SOLVCON  to  VTK. 
a  Use  VTK's  official  Python  wrappers  to  access  all  VTK  functionalities, 
e.g.,  cut  surface,  contour,  iso-surface,  etc. 
a  In  situ  visualization  is  programmed  in  driving  scripts, 
a  Visualization  differs  from  one  case  to  another, 
a  No  hard-wired  code. 

a  The  driving  scripts  become  an  application  program  that  can  deliver 
analyzed  results  including  graphic  files. 
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Open-Source  Practice 

SOLVCON  is  Open-Sourced 


®  Released  under  GNU  GPL  v2  with  full  source. 

a  Freely  available  at  http://solvcon.net/ 
a  Systematic  open-source  practices:  (i)  Distributed  version  control 
system,  (ii)  unit  testing,  (iii)  issue  tracking,  (iv)  continuous 
integration,  (v)  auto-generated  API  documentation. 
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Python -Based  Super  computing  Framework  for  Multi -Physics  Conservation  Laws 

1  I  Install  &  Use  Technology  API  Email  Us 

SOLVCON  is  a  software  framework  for  solving  hyperbolic 

NEWS 

partial  differential  equations  (PDEs).  SOLVCON  uses  the 

space -time  Conservation  Element  and  Solution  Element 

fCESE")  method  for  time-accurate  solutions.  The  goal  of 

released.  This  release  starts  to 

SOLVCON  is  to  enable  high-fidelity  solutions  for  design 

support  incenters  for  the  CESE 
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